Automated fluorescence imaging and single cell segmentation

ABSTRACT

Systems and methods for automated, non-supervised, parameter-free segmentation of single cells and other objects in images generated by fluorescence microscopy. The systems and methods relate to both improving initial image quality and to improved automatic segmentation on images. The methods will typically be performed on a digital image by a computer or processor running appropriate software stored in a memory.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a Divisional of U.S. patent application Ser. No.17/193,988, filed Mar. 5, 2021, which claims the benefit of U.S.Provisional Patent Application Ser. No. 62/985,539, filed Mar. 5, 2020.The entire disclosure of all the above documents is herein incorporatedby reference.

BACKGROUND OF THE INVENTION Field of the Invention

This disclosure is related to the field of analysis of digital imagesand particularly to automated analysis of digital images generated byfluorescence microscopy to perform single cell or other objectsegmentation on those images.

Description of the Related Art

Flow cytometers (FCM) and fluorescence-activated cell sorters (FACS) arecurrently some of the primary tools for the characterization of cells.Both systems utilize fluorescent antibodies or other fluorescing probes(fluorophores) to tag cells having particular characteristics ofinterest and then detect the fluoresced light to locate the targetcells. Both systems are widely used in both biomedical research andclinical diagnostics and can be used to study physical properties (forexample, cell size and shape) and biochemical properties (for example,cell cycle distribution and DNA contents) of cells. Flow cytometry hasalso become a valuable clinical tool to monitor the progression ofcertain cell diseases.

Information about the cells of interest will typically be obtainedoptically. FCM/FACS involves the introduction of specific tagged probes,which fluoresce at a certain wavelength when exposed to a certain lightinput, and the connection of the tagged probe to a target of interestallows for the identification of individual cells to which the probe hasattached. It should be recognized that in FCM/FACS, the measured signalis not clean. The fluorescence is typically detected optically by adevice such as a photomultiplier tube (PMT) and the PMT receivesbackground signal (light pollution) and autofluorescence of the cell inaddition to light from the markers. Further, even light specificallyfrom the markers can be obtained from non-specifically bound markerspresent in the sample. This is in addition to the specifically boundmarkers attached to target molecules on or in the cells, which is thevalue of interest.

For determining the desired result, only the signal of the specificallybound markers is of relevance and the rest of the measured signal isconsidered “noise” or error. This error inhibits the specific signalfrom being measured as sensitively as is often desired and in cases of alow intensity of the specific signal and a relatively high intensity ofthe noise, the specific signal may “disappear” in the noise leading tothe false negative results. The amount of noise (signal-to-noise ratio),therefore, influences the detection sensitivity of the measurement.Further, the noise can also influence the distinction between positiveand negative cell populations, as the measured positive and negativevalues may become indistinguishable.

The main reasons for measurement errors are background signals(pollution) within the small dynamic range of the detection device, highautofluorescence signals of the cells, unspecific binding of the markersto the cell, and variations that occur during the staining process (e.g.different marker concentrations, labeling of the marker or stainingconditions). While it is desirable to reduce the unspecific backgroundsignals as much as possible, FCM/FACS-based methods have only limitedpotential to reduce background signals. In principle, it is onlypossible to reduce the basic background signal of the FCM/FACS device byavoiding the accumulation or detection of interfering signals.

In order to eliminate the background signal of the detection device andthe autofluorescence of the cells, currently known methods typically usea control sample in which the investigated cells are not treated withthe fluorophore and the values of the background and autofluorescencefrom this sample are subtracted from the actual measurement runs.However, this approach has a number of drawbacks. By comparing twodifferent populations of cells, additional measurement errors oftenoccur. This is due to the fact that the populations may differ in theirdensity, age, expression intensity of the investigated marker, etc.Further, also the fluorescence of the sample material (aside from thecells) may vary between different samples due to variations in theirmanufacturing processes or their compositions. Finally, and often mostsignificantly, the control sample value is typically an average valuethat is calculated on the basis of a control sample measurement and doesnot take into consideration the variations that exist between individualcells.

One of the major issues in FACS, and the analysis of digital images ofbiological systems more generally, is the need for segmentation.Segmentation typically refers to the need to identify of boundaries ofcells within a digital image generated of the cells. Once obtained,digital images such as those generated with the use of fluorophores needto be segmented so that individual cells can be identified, for example,so they can be counted. Segmentation is typically performed using analgorithm. A watershed transform is one image processing technique thathas been used for segmenting images of cells. With the watershedtransform, a digital image may be modeled as a three-dimensionaltopological surface, where intensity values of pixels representgeographical heights. Thus, items associated with more of thefluorophore (which may, for example, be the nucleus of a cell) areidentified as peaks or “volcanos” while cell walls would be in valleysbetween peaks.

A problem with many segmentation algorithms, however, is variations inthe histology of different tissue types and cells. Thus, certainalgorithms may not produce an accurate segmentation without adaptationor training to the specific type of cell. This difference can also maketraining automated segmentation systems very difficult. Algorithms canfail both directions causing an image to be over-segmented (indicating acomplete cell where only a portion of a cell is present) orunder-segmented (indicating a single cell when there are multiple cellspresent).

To try and combat the problem, alternative algorithms have been proposedincluding those discussed in U.S. Pat. No. 8,995,740, the entiredisclosure of which is herein incorporated by reference. These systems,however, often still have problems due to underlying image capturequality where even the best algorithms cannot operate to their fullestpotential, and even then they often still have trouble with topologicalstructures (such as “atolls”) which can be common in certain cellimages.

This problem becomes particularly acute because training methods forautomated analysis of cells are typically trained on a wide variety ofheterogeneous cells. Thus, cells having a low autofluorescence and cellshaving a high autofluorescence are typically calibrated against the sameaverage control sample value (or training data), with the result thatcells having a low autofluorescence are prone to be evaluated as falsenegatives and cells having a high autofluorescence are likely to beevaluated as false positives. As all the above can significantly impairsensitivity of the detection method, it would be desirable to havemethods available that overcome the drawbacks of existing methods andallow a more sensitive and reliable cell analysis.

U.S. patent application Ser. No. 15/708,221, the entire disclosure ofwhich is herein incorporated by reference, provides for systems andmethods that can be used to cancel autofluorescence and background noisemore effectively. Specifically, this is performed by determining theautofluorescence not from a generalized sample or training selection ofmaterials, but from the actual sample which is to be tested. Thisprovides for substantial improvement in signal-to-noise ratio. However,those systems and methods do not provide for advanced calibrationtechniques utilizing a high dynamic ranging imaging (HDRI) camera and donot provide for advanced processing techniques that can provideincreased accuracy in cell segmentation when images have been soprepared.

SUMMARY OF THE INVENTION

The following is a summary of the invention in order to provide a basicunderstanding of some aspects of the invention. This summary is notintended to identify key or critical elements of the invention or todelineate the scope of the invention. The sole purpose of this sectionis to present some concepts of the invention in a simplified form as aprelude to the more detailed description that is presented later.

Because of these and other problems in the art, there is describedherein, among other things, a method for calibrating an imaging systemfor cellular imaging, the method comprising: providing an imaging systemfor imaging cells which have tagged with a fluorophore having a range offluorescing wavelengths; performing a calibration for autofluorescencecomprising: providing the imaging system with a sample of unstainedcells having a range of autofluorescing wavelengths; illuminating thesample with a source of illumination; imaging the sample across adynamic range including all of the range of fluorescing wavelengths andall of the range of autofluorescing wavelengths; performing acalibration for chromatic aberration comprising: providing the imagingsystem with a sample of cells; illuminating the sample with a source ofillumination; obtaining a first image of the sample of cells; altering aposition of the imaging system relative to the sample of cells; andobtaining a new image of the sample of cells.

In an embodiment, of the method, the method is performed by a digitalcomputer including memory with instructions for performing the method

In an embodiment of the method, a minimum photon acquisition time of theimaging system is set separately for each wavelength within the dynamicrange; and the minimum photon acquisition time is sufficient fordetection of all values within the complete dynamic range.

In an embodiment of the method, the dynamic range comprises allwavelengths the imaging system images.

In an embodiment of the method, the imaging system comprises a digitalgrayscale camera.

In an embodiment of the method, the camera is provided with a filtersetbased on the fluorophore.

In an embodiment of the method, the camera is a High Dynamic RangeImaging (HDRI) camera.

In an embodiment of the method, the camera generates High Dynamic Range(HDR) by exposure fusion to provide for improved contrast.

In an embodiment of the method, the cells which have tagged with thefluorophore are a different type of cells to the unstained cells

In an embodiment of the method, the sample of cells is a different typeof cells to the sample of unstained cells.

In an embodiment of the method, the sample of cells is the sample ofunstained cells.

There is also described herein, in an embodiment, a method for analyzinga cellular image, the method comprising: providing a sample of cellswhich have been tagged with a fluorophore having a range of fluorescingwavelengths; illuminating the sample with a source of illumination;imaging the illuminated sample over the range of fluorescing wavelengthsto produce a sample image; subtracting a calibration image from thesample image to produce a calibrated image; representing the image as atopological curve, a height of the curve at each curve pixelrepresenting an intensity of fluorescence at an image pixel in thecalibrated image; searching the topological curve for a pixel groupinghaving height above a selected height; choosing a new height, lower thanthe selected height; repeating the searching using the new height as theselected height; and for each identified pixel grouping: following aconvex border of the topological curve away from the pixel group to aninflection point where inflexion is decreasing at a convex side; andidentifying the convex border including the pixel group as a cell in thecalibrated image.

In an embodiment, of the method, the method is performed by a digitalcomputer including memory with instructions for performing the method

In an embodiment of the method, the calibration image is formed by amethod comprising: providing the imaging system with a sample ofunstained cells having a range of autofluorescing wavelengths;illuminating the sample with a source of illumination; and imaging thesample across a dynamic range including all of the range of fluorescingwavelengths and all of the range of autofluorescing wavelengths;

In an embodiment of the method, a minimum photon acquisition time of theimaging system is set separately for each wavelength within the dynamicrange; and the minimum photon acquisition time is sufficient fordetection of all values within the complete dynamic range.

In an embodiment of the method, the dynamic range comprises allwavelengths the imaging system images.

In an embodiment, the method, further comprises: before obtaining thesample image, performing a calibration for chromatic aberration on theimaging system, the calibration comprising: illuminating the sample ofcells with a source of illumination; obtaining a first image of thesample of cells from the imaging system in a first position; moving theimaging system to a second different position relative to the sample ofcells; obtaining a second image of the sample of cells; and whengenerating the sample image: positioning the imaging system in the firstposition if the first image is optimized compared to the second image;and placing the imaging system if it is not.

In an embodiment of the method, the imaging system comprises a digitalgrayscale camera provided with a filterset based on the fluorophore.

In an embodiment of the method, the imaging system comprises a HighDynamic Range Imaging (HDRI) camera.

In an embodiment of the method, the imaging system generates HighDynamic Range (HDR) by exposure fusion to provide for improved contrast.

In an embodiment of the method, the calibrated image is used to train aneural network.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flowchart of an embodiment of a method for automatedimage processing which may be used to prepare images for cellsegmentation.

FIG. 2 shows a sensitivity curve of a common CCD chip.

FIG. 3A shows a topological representation of two adjacent cells withthe image acquisition calibrated in accordance with the firstcalibration.

FIG. 3B shows a topological representation of the two adjacent cells ofFIG. 3A without the image acquisition calibrated in accordance with thefirst calibration.

FIG. 4A shows a difficult-to-segment area of an FCM imaged with astandard low dynamic range camera.

FIG. 4B shows the difficult-to-segment area of FIG. 4A imaged with ahigh dynamic range imaging (HDRI) camera.

FIG. 5A shows a histogram of a low dynamic range camera. Specifically amicroscope camera with 8 bits of dynamic range.

FIG. 5B shows a histogram of a high dynamic range imaging (HDRI) cameraon the same image as FIG. 5A.

FIG. 6A shows a liver section stained for nuclei prior to thesubtraction of the corresponding autofluorescence image.

FIG. 6B shows the liver section of FIG. 6A after the subtraction of thecorresponding autofluorescence image.

FIG. 7 shows a portion of an image which is particularly hard to segmentdue to variability in shape and staining intensity.

FIG. 8 shows a 2D topological curve showing the identification of seedsand cells with different “water levels” utilizing an inverse watershedmethodology.

FIG. 9 shows a portion of an image with a seed identified.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 provides a flowchart of an embodiment of a method for automated,non-supervised, parameter-free segmentation of single cells and otherobjects in images generated by fluorescence microscopy. This method willtypically be performed on a digital image by a computer or processorrunning appropriate software stored in a memory. However, in analternative embodiment, the method may be implemented throughelectromechanical hardware, such as but not limited to circuitry. Thesystems and methods will typically be provided as part of the operationof, or with, an automated microscope with a digital image sensor like acharge couple device (CCD) or complementary metal oxide semiconductor(CMOS). The computer in combination with such software or the hardwareso designed comprises an embodiment of a system of the present inventionas do such elements in conjunction with other elements of cell or otherbiological tissue analysis.

Throughout this disclosure, the term “computer” describes hardware thatgenerally implements functionality provided by digital computingtechnology, particularly computing functionality associated withmicroprocessors. The term “computer” is not intended to be limited toany specific type of computing device, but it is intended to beinclusive of all computational devices including, but not limited to:processing devices, microprocessors, personal computers, desktopcomputers, laptop computers, workstations, terminals, servers, clients,portable computers, handheld computers, smart phones, tablet computers,mobile devices, server farms, hardware appliances, minicomputers,mainframe computers, video game consoles, handheld video game products,and wearable computing devices including, but not limited to eyewear,wrist wear, pendants, and clip-on devices.

As used herein, a “computer” is necessarily an abstraction of thefunctionality provided by a single computer device outfitted with thehardware and accessories typical of computers in a particular role. Byway of example and not limitation, the term “computer” in reference to alaptop computer would be understood by one of ordinary skill in the artto include the functionality provided by pointer-based input devices,such as a mouse or track pad, whereas the term “computer” used inreference to an enterprise-class server would be understood by one ofordinary skill in the art to include the functionality provided byredundant systems, such as RAID drives and dual power supplies.

It is also well known to those of ordinary skill in the art that thefunctionality of a single computer may be distributed across a number ofindividual machines. This distribution may be functional, as wherespecific machines perform specific tasks; or, balanced, as where eachmachine is capable of performing most or all functions of any othermachine and is assigned tasks based on its available resources at apoint in time. Thus, the term “computer” as used herein, may refer to asingle, standalone, self-contained device or to a plurality of machinesworking together or independently, including without limitation: anetwork server farm, “cloud” computing system, software-as-a-service, orother distributed or collaborative computer networks.

Those of ordinary skill in the art also appreciate that some devicesthat are not conventionally thought of as “computers” neverthelessexhibit the characteristics of a “computer” in certain contexts. Wheresuch a device is performing the functions of a “computer” as describedherein, the term “computer” includes such devices to that extent.Devices of this type include but are not limited to: network hardware,print servers, file servers, NAS and SAN, load balancers, and any otherhardware capable of interacting with the systems and methods describedherein in the matter of a conventional “computer.”

Throughout this disclosure, the term “software” refers to code objects,program logic, command structures, data structures and definitions,source code, executable and/or binary files, machine code, object code,compiled libraries, implementations, algorithms, libraries, or anyinstruction or set of instructions capable of being executed by acomputer processor, or capable of being converted into a form capable ofbeing executed by a computer processor, including without limitationvirtual processors, or by the use of run-time environments, virtualmachines, and/or interpreters. Those of ordinary skill in the artrecognize that software may be wired or embedded into hardware,including without limitation onto a microchip, and still be considered“software” within the meaning of this disclosure. For purposes of thisdisclosure, software includes without limitation: instructions stored orstorable in RAM, ROM, flash memory BIOS, CMOS, mother and daughter boardcircuitry, hardware controllers, USB controllers or hosts, peripheraldevices and controllers, video cards, audio controllers, network cards,Bluetooth® and other wireless communication devices, virtual memory,storage devices and associated controllers, firmware, and devicedrivers. The systems and methods described here are contemplated to usecomputers and computer software typically stored in a computer- ormachine-readable storage medium or memory.

Throughout this disclosure, terms used herein to describe or referencemedia holding software, including without limitation terms such as“media,” “storage media,” and “memory,” may include or excludetransitory media such as signals and carrier waves.

Throughout this disclosure, the term “real-time” generally refers tosoftware performance and/or response time within operational deadlinesthat are effectively generally cotemporaneous with a reference event inthe ordinary user perception of the passage of time for a particularoperational context. Those of ordinary skill in the art understand that“real-time” does not necessarily mean a system performs or respondsimmediately or instantaneously. For example, those having ordinary skillin the art understand that, where the operational context is a graphicaluser interface, “real-time” normally implies a response time of aboutone second of actual time for at least some manner of response from thesystem, with milliseconds or microseconds being preferable. However,those having ordinary skill in the art also understand that, under otheroperational contexts, a system operating in “real-time” may exhibitdelays longer than one second, such as where network operations areinvolved which may include multiple devices and/or additional processingon a particular device or between devices, or multiple point-to-pointround-trips for data exchange among devices. Those of ordinary skill inthe art will further understand the distinction between “real-time”performance by a computer system as compared to “real-time” performanceby a human or plurality of humans. Performance of certain methods orfunctions in real-time may be impossible for a human, but possible for acomputer. Even where a human or plurality of humans could eventuallyproduce the same or similar output as a computerized system, the amountof time required would render the output worthless or irrelevant becausethe time required is longer than how long a consumer of the output wouldwait for the output, or because the number and/or complexity of thecalculations, the commercial value of the output would be exceeded bythe cost of producing it.

The definitions provided in U.S. patent application Ser. No. 15/708,221are also relevant to the discussion herein and those definitions arespecifically incorporated by reference as definitions for those terms asused herein.

The present image analysis will typically be performed by the computerin real-time so as to allow the analysis results to be readily useablein both research and diagnostic or clinical settings and for theselection of treatment for disease indicated via the image analysis.Further, the process is typically automated so that the image analysiscan be performed either with minimal or with no human intervention.Specifically, the image analysis is typically performed with a computerperforming the acts of obtaining images and/or evaluating the images forsegmentation of cells within the image without a human being needing toassist in the analysis. Most of the time, the automated systems will befurther combined with other automated systems which can utilize thesegmented image to provide further evaluation of the cells of interestto a human user, but this is by no means required.

The present systems and methods, in an embodiment, combine systems andmethods for improving both the process of image acquisition (forproviding improved input data for the image processing step) and imageprocessing in the nature of cell segmentation. However, one of ordinaryskill in the art would recognize that the image acquisition discussedherein could be used to provide improved images to traditional imageprocessing systems and methods and that traditional image acquisitionsystems could also be used to provide data to the present imageprocessing systems in alternative embodiments.

Image processing elements discussed herein are typically designed tohandle a high heterogeneity of input data related to cell segmentation.Heterogeneity within the samples can be caused by, for example,different sample sources of the underlying cells (e.g., brain, liver,spleen), sample quality, staining quality, or other factors. In otherembodiments, the systems and methods herein can be used to evaluateimages directed to things other than cells which may have high or lowheterogeneity. For example, the systems for image acquisition discussedherein can be used to improve signal-to-noise ratio withfluorescence-based images of virtually anything. Further, segmentationof images need not provide for detection of individual biological cells,but may be used to detect other subcomponents of an image wheresegmentation is determined to be useful. However, for ease ofdiscussion, the present disclosure will utilize as an exemplaryembodiment the imaging of cells (specifically biological cells) andsegmenting of those images to detect individual cells.

As shown in FIG. 1 , the method and operation of the system typicallyfirst comprises obtaining a sample of cells which will typically beimmobilized on a solid substrate (101). This may be performed by anymethod known to those of ordinary skill in the art and all such knownmethods are incorporated herein by reference. U.S. patent applicationSer. Nos. 15/708,221 and 13/126,116, the entire disclosures of which areherein incorporated by reference, provide examples of embodiments of howsuch immobilization and sample preparation may be performed.

Once the sample is obtained (101), the sample is exposed to an imageacquisition system (103). The image acquisition system will typically bea digital grayscale camera and the camera will typically be providedwith a filterset which will be selected based on the fluorophore to beused in the imaging. The camera will typically be of the form of a HighDynamic Range Imaging (HDRI) camera or a software algorithm thatcontrols a camera in a way that it generates HDR by exposure fusion toprovide for improved contrast. The systems and methods discussed hereincan utilize a camera wherein HDRI is provided using multiple exposuresor with a single exposure depending on the camera selected and thespecific embodiment of the systems and methods being utilized.

The image acquisition system may first be calibrated (105) withtypically two calibration actions (201) and (203). However, calibration(105) does not have to be performed necessarily during or coextensivelywith image acquisition (107). Alternatively, calibration (105) canalternatively be performed during setup or maintenance of the system,each day before starting imaging, or when needed. Thus, a decision (104)may be made to calibrate (105) the system or proceed directly to imageacquisition (107). Further, calibration (105) may be performed withoutsubsequent image acquisition (107) in which scenario the system wouldtypically cease operation after element (203).

In the first calibration (201), the image acquisition system iscalibrated against the autofluorescence of unstained cells as well asany inherent background signal utilizing any light sources that will beused in the actual imaging runs against a stained sample. In anembodiment, the minimum photon acquisition time will be set separatelyfor each wavelength within the complete dynamic range ofautofluorescence plus the fluorescently labeled detectors to provide fora signal which is sufficient for detection of all values. To put thisanother way, the dynamic range of the imaging can be chosen to insure itincludes any autofluorescence or background signal detected at anywavelength detectable by the camera as well as the specific wavelengthsfor the fluorophore selected for staining.

This type of first calibration (201) is generally used to compensate forthe differences in sensitivity of the image acquisition system withregards to different wavelengths that it can detect and which may bepresent. The first calibration will typically allow for a determinationof what is background and autofluorescence for the present sample basedon current measurement and, therefore, this can reduce reliance onsignal size to determine if signals are of interest or are noise.

As an example, FIG. 2 provides the sensitivity curve (401) of a commonCCD chip which may be used in the image acquisition system. In the chipof FIG. 2 , at lower and higher wavelengths, longer exposure times maybe used to assure that the lower limit of the dynamic range of the imageacquisition system is capable of capturing the autofluorescence of anycell present in the sample within the dynamic range of the camera foreach used acquisition wavelength. It should be recognized that anyparticular imaging run of the system or method may utilize any or all ofthe available range of the camera. This first calibration (201) willtypically be run at all wavelengths (or at selected subgroupings ofwavelengths depending on sensitivity and time constraints) that are tobe used in the acquisition run of this particular sample which isreferred to as the dynamic range.

In the second calibration (203), the image acquisition system iscalibrated for chromatic aberration. Chromatic aberration (which iscommonly called the “rainbow effect”) leads to projection of photonsfrom a single position on the specimen to different locations on thecamera dependent on the wavelength of the photon and lens and/or chipdesign. When not compensated, the x-. y-, and z-axis offsets betweendifferent wavelengths leads to blurring of the image or offset thuslowering the quality of input data for the segmentation process. Thesecond calibration (203) will typically be performed by takingnon-calibrated images in a first round. These images may be taken ofstained or unstained samples depending on embodiment. In a second andfollowing rounds, the x, y, and z positioning of the imager is changedleaving the sample constant. The position movement is typically in astepwise or likewise repeatable manner to find an optimal x, y, and zoffset between the single filtersets used for the different wavelengths.This calibration will then be used during the actual imaging runs.

FIGS. 3A and 3B show an example of how the second calibration (203) canimprove image quality. FIG. 3A shows two adjacent cells (501) and (503)as peaks in a topological representation (staining intensity as height)(601) with the image acquisition system calibrated as contemplated inthe second calibration. FIG. 3B shows the same two adjacent cells (501)and (503) without the second calibration having been performed whichshows blurring due to offset imaging. As should be apparent, theblurring results in it being more difficult to differentiate the twodifferent cells.

After the second calibration (203) is complete, the image acquisitionsystem will typically be considered calibrated and ready to begin imageacquisition (107). Image acquisition (107) will typically comprise atleast two and possibly more acquisition passes and then correction (109)of the images to remove artifacts from autofluorescence and backgroundsignals. The first acquisition pass (301) will generally be done on asample before staining.

The first acquisition pass (301) will typically be performed at adynamic range that covers the complete dynamic range of autofluorescencein every imaged wavelength and, therefore, through the entire dynamicrange of the camera which has been selected for this imaging. As shouldbe apparent, this dynamic range will generally correspond to the dynamicrange over which the first calibration (201) was performed. The firstimage acquisition (301) will also typically be performed with the sameinput light or lights that will be used in conjunction with the lateracquisitions.

After this first pass (301) is completed, the sample will be exposed tothe fluorophore (which is typically in the form of a detection conjugatehaving a binder portion and a fluorochrome portion) and stained asdesired. The second pass (303) will involve essentially repeating thesteps of the first image acquisition pass on the now stained sample.Acquisition will typically be performed over a dynamic range that coversthe complete dynamic range of autofluorescence plus biomarker expressionrevealed by the fluorescently labeled detectors (e.g. antibodies,aptamers). It should be recognized that for simplicity the dynamic rangeof both the first image acquisition pass (301) and the secondacquisition pass (303) may be the same, however, the first imageacquisition pass (301) may utilize a smaller dynamic range since it isonly looking for autofluorescence.

FIGS. 4A and 4B show an example of a difficult-to-segment area takenwith a low dynamic range camera in FIG. 4A and an HDRI camera in FIG.4B. Histograms generated from the images of FIGS. 4A and 4B are shown inFIGS. 5A and 5B reveal that only then HDRI images of FIG. 4B can coverthe complete dynamic range of the fluorescence emitted by the sample,whereas standard microscope cameras with standard 8-bit dynamic rangecan only capture about 200 values of dynamic range. This lack of dynamicrange in FIG. 4A results in the generation of artifacts and a less clearimage which reduces the ability to segment cells in the image.

After the second image acquisition (303) is complete, the first passimage (301) will typically be subtracted from the second (and any othersubsequent) pass image (303) in the correction (109). A methodology fordoing this is discussed in the above referenced U.S. patent applicationSer. No. 15/708,221 and is incorporated herein by reference. Thesubtraction serves to help eliminate illumination artefacts of theoptical system appearing inside the image and also reducesautofluorescence signals obfuscating the signals generated by thestaining itself. FIG. 6A shows a liver section stained for nuclei priorto correction (109), and FIG. 6B shows the same section with correction(109) having been performed.

After the completion of the correction (109), The image is typicallyconsidered optimized for segmentation of single objects or cells.Segmentation (111) may be performed using any system or method known toa person having ordinary skill in the art. However, due to the very highvariability of object shapes, sample, and staining quality, a robustalgorithm or machine learning approach will typically be preferred toassure a high sensitivity and specificity of object recognition. FIG. 7shows a particularly hard to segment image due to variability of shapeand staining intensity.

In an embodiment, calibrated and optimized high dynamic range inputimage data may be obtained from a wide variety of sources utilizing thesystems and methods discussed in FIG. 1 through the correction (109).These images may then be used to train a neural network (or similarprocessing system) with supervised learning and backpropagation torecognize single objects and object borders in the manner known to thosehaving ordinary skill in the art. The images provided may be fromheterogeneous sources with a wide array of dynamic ranges andfluorescence properties as the generation of such high quality imagesassists in the elimination of artifacts and differences across thesources. Alternatively, the neural network (or similar) may be providedwith images from more homogenous sources (e.g. only one type of cell) ifthat is desired. This neural network may then be used to performsegmentation (111) in an embodiment.

In the embodiment of FIGS. 7 and 8 , the segmentation (111) is performedon the image using a form of inverse watershed transform which isreferred to herein as cyclic seed detection or a “falling water level”analysis. Seed detection in conjunction with traditional watershedtransform is discussed in, for example, U.S. Pat. No. 8,995,740 theentire disclosure of which is herein incorporated by reference.

In the cyclic seed detection system and method used herein, topologicalanalysis is performed as in a watershed transform where the intensity offluorescence is used to indicate a “height” of each pixel and thereforepeaks in the topology will typically indicate a target element of arelevant cell (for example the cell nucleus). This is as is contemplatedin FIGS. 3A and 3B above as well. Watershed transform, however, is proneto error for “atoll” or “erupted volcano” like objects (this could be anobject such as seed (811) in FIG. 8 ). Objects such as these can begenerated by staining quality or self-quenching, for example. In effect,traditional watershed transform requires perfect “volcano” shapes towork.

Inverse watershed transform as contemplated herein serves to removewater slowly instead of the traditional watershed transform which floodsthe topology with water. FIG. 8 provides for an exemplary 2D topologicalcurve from a portion of an image to illustrate the operation of theprocess of cyclic seed detection or inverse watershed transform. Tobegin with, a first intensity is selected which is the “high waterlevel” (801). Any pixel grouping which is identified as being above thisintensity (801) is provided as a seed (811) and will be identified asbelonging to an individual cell. FIG. 9 provides for an example of aseed (711) within an image.

Returning to FIG. 8 , a second lower intensity (803) is then selectedand the process is repeated to identify additional seeds (813) whichhave now risen “above” the new “water level” (803) of this lowerintensity (803). This lowering of the intensity and searching for newseeds will be repeated with new lower levels typically selected in aconsistent stepwise fashion. In the embodiment of FIG. 8 , twoadditional levels (805) and (807) are selected and each reveals a newseed (815) and (817). FIG. 8 also illustrates level (809) which wouldcorrespond to the initial reading on a more traditional watershedtransform. This level would generate a single seed (819) instead of thethree seeds (811), (813), and (815).

From each of these seeds (811), (813), (815) and (817) there is presumedto be an associated cell whose boundaries need to be detected. Thus, theseed can be used as a source to determine the extension of the cellbelonging to the seed (811), (813), (815), and (817). This is typicallycarried out by following the convex border of the topological curve awayfrom the seed (811), (813), (815) and (817) to try and find the outerrim of the “isle” by looking for inflection points where the inflexionis decreasing at the convex side. In FIG. 8 , this results in the leftportion of the image being segmented into the four target cells.

While the invention has been disclosed in connection with certainembodiments, this should not be taken as a limitation to all of theprovided details. Modifications and variations of the describedembodiments may be made without departing from the spirit and scope ofthe invention, and other embodiments should be understood to beencompassed in the present disclosure as would be understood by those ofordinary skill in the art.

It will further be understood that any of the ranges, values,properties, or characteristics given for any single component of thepresent disclosure may be used interchangeably with any ranges, values,properties, or characteristics given for any of the other components ofthe disclosure, where compatible, to form an embodiment having definedvalues for each of the components, as given herein throughout. Further,ranges provided for a genus or a category may also be applied to specieswithin the genus or members of the category unless otherwise noted.

Finally, the qualifier “generally,” and similar qualifiers as used inthe present case, would be understood by one of ordinary skill in theart to accommodate recognizable attempts to conform a device to thequalified term, which may nevertheless fall short of doing so. This isbecause terms such as “circular” are purely geometric constructs and noreal-world component is a true “circle” in the geometric sense.Variations from geometric and mathematical descriptions are unavoidabledue to, among other things, manufacturing tolerances resulting in shapevariations, defects and imperfections, non-uniform thermal expansion,and natural wear. Moreover, there exists for every object a level ofmagnification at which geometric and mathematical descriptors fail dueto the nature of matter. One of ordinary skill would thus understand theterm “generally” and relationships contemplated herein regardless of theinclusion of such qualifiers to include a range of variations from theliteral geometric meaning of the term in view of these and otherconsiderations.

1. A method for analyzing a cellular image, the method comprising:providing a sample of cells which have been tagged with a fluorophorehaving a range of fluorescing wavelengths; illuminating said sample witha source of illumination; imaging said illuminated sample over saidrange of fluorescing wavelengths to produce a sample image; subtractinga calibration image from said sample image to produce a calibratedimage; representing said image as a topological curve, a height of saidcurve at each curve pixel representing an intensity of fluorescence atan image pixel in said calibrated image; searching said topologicalcurve for a pixel grouping having height above a selected height;choosing a new height, lower than said selected height; repeating saidsearching using said new height as said selected height; and for eachidentified pixel grouping: following a convex border of said topologicalcurve away from said pixel group to an inflection point where inflexionis decreasing at a convex side; and identifying said convex borderincluding said pixel group as a cell in said calibrated image.
 2. Themethod of claim 1 wherein said calibration image is formed by a methodcomprising: providing said imaging system with a sample of unstainedcells having a range of autofluorescing wavelengths; illuminating saidsample with a source of illumination; and imaging said sample across adynamic range including all of said range of fluorescing wavelengths andall of said range of autofluorescing wavelengths.
 3. The method of claim2 wherein: a minimum photon acquisition time of said imaging system isset separately for each wavelength within said dynamic range; and saidminimum photon acquisition time is sufficient for detection of allvalues within said complete dynamic range.
 4. The method of claim 2wherein said dynamic range comprises all wavelengths said imaging systemimages.
 5. The method of claim 1 further comprising: before obtainingsaid sample image, performing a calibration for chromatic aberration onsaid imaging system, said calibration comprising: illuminating saidsample of cells with a source of illumination; obtaining a first imageof said sample of cells from said imaging system in a first position;moving said imaging system to a second different position relative tosaid sample of cells; obtaining a second image of said sample of cells;and when generating said sample image: positioning said imaging systemin said first position if said first image is optimized compared to saidsecond image; and placing said imaging system if it is not.
 6. Themethod of claim 1 wherein said imaging system comprises a digitalgrayscale camera provided with a filterset based on said fluorophore. 7.The method of claim 1 wherein said imaging system comprises a HighDynamic Range Imaging (HDRI) camera.
 8. The method of claim 1 whereinsaid imaging system generates High Dynamic Range (HDR) by exposurefusion to provide for improved contrast.
 9. The method of claim 1wherein said calibrated image is used to train a neural network.